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ABSTRACT 



This study compared the benefits and costs associated with 
online versus paper-and-pencil surveys for a random sample of undergraduate 
students. Students completed the College Selection Survey, which had them 
evaluate the soundness of 10 reasons for choosing, or not choosing, to attend 
the university, if they could make their choice over again. Half of the 
sample was contacted by mail and asked to complete the paper-and-pencil 
survey. The other half was contacted by e-mail and asked to complete the same 
survey online. Willing students participated in focus groups to discuss how 
influential the survey reasons would be if they could select their college 
over again. The overall response rate was 33.3 percent for the 
paper-and-pencil sample and 26.2 percent for the online sample. A higher 
percentage of traditional age than nontraditional students responded online. 
Students in the two groups had similar feelings about whether they would 
choose the institution again, with 66.7 percent saying yes, 8.9 percent 
saying no, and 24.3 percent undecided. Students believed that the main 
advantage to online surveying was its convenience, though students without 
computers considered it difficult to find open computers in the campus 
laboratories. (Contains 15 references.) (SM) 
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On-Line Vs. Paper-And-Pencil Surveying of Students: A Case Study 



In the fall of 1999, a mid-size, Southeastern, public university administered the College 
Selection Survey to a random sample of students in either an on-line or paper-and-pencil format 
Analysis was conducted viewing the differences in responses by method of administration. 

By using both on-line and paper-and-pencil versions of the same survey the authors 
examine the benefits and costs associated with administering surveys on the World Wide Web. 
Findings from this study should provide a clearer picture of the factors involved in conducting 
on-line surveys and aid institutional researchers in deciding whether or not on-line surveys are 
appropriate for their institutions. 
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On-Line Vs. Paper-And-Pencil Surveying of Students: A Case Study 



1 



Two major developments in the world of institutional research have instigated significant 
changes in the way institutional research professionals work. One development is the enhanced 
accountability felt by institutions of higher education nationwide. As a result, outcomes 
assessment, surveying in particular, has become an increasingly major component of institutional 
researchers’ activities. 

The second development is the Internet and the World Wide Web. The wide access, ease 
of use, and ease of analysis provided by the World Wide Web has made surveying on-line a very 
attractive alternative to traditional, paper-and-pencil surveys. As a result, the office of 
Institutional Research staff at a mid-size, public university has begun to utilize the Web as a part 
of its surveying work. With this move in institutional research toward Web-based research, the 
need to analyze this method of survey administration versus more conventional methods is in 
order. 

As stated, part of the reason for the increased focus on surveying via the Web is due to 
the many advantages this medium offers. A frequently cited advantage is the cost associated 
with on-line research. Schmidt (1997) finds surveying on-line has benefits including “savings in 
both time and money for survey researchers” (p. 274). Among the savings found are the 
reduction or elimination of paper resources and elimination of data entry expense and time 
(Schmidt, 1997, p. 275). Gjestland (1996) notes “because all interviews on the Internet are self- 
administered, there is a tremendous cost savings in not having interviewers” (p. 3). Gaddis 
reports (1998) that the “relatively low cost of designing and conducting an on-line survey” (p. 

67) increases its appeal to researchers. 
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Other advantages of on-line surveying have been reported. One such benefit is that 
researchers, when using the Internet to conduct surveys, are not bound by geography (Gjestland, 
1996; Smith & Leigh, 1997; Swoboda, Miihlberger, Weitkunat, Schneeweiss, 1997). Smith and 
Leigh (1997) praise the accessibility of the Internet as well as its potential to transmit a variety of 
communications including text, images, and sound (p. 496). Davis (1997) agrees with the 
conveniences or ease of access and adds the benefit of “confidentiality of responses” (p. 2). 

Archiving of data resulting from on-line surveys is also a stated benefit (Smith & Leigh, 
1997; Schmidt, 1997). Several studies applaud the rapid return of instruments when published 
on-line, which leads to more rapid analysis (Gaddis, 1998; Gjestland, 1996; Swoboda et al., 
1997;). Gaddis indicates (1998) that when relying on the Web for surveying, one is also less 
likely to have “interference from people or institutions that may prevent a survey from being 
delivered to a respondent” (p.67). 

Davis (1997) sums up these points by stating that “Internet surveys are fast, effective, and 
cost-efficient” (p. 1). 

With savings on cost and the ease of survey processing, the remaining questions are: 

(1) is the profile of the Web respondent different, (2) do response rates differ, (3) are the results 
obtained from those responding via electronic formats different from those filling out a pencil- 
and-paper version. 

Considering first the profile of Web respondents, apprehensions frequently are voiced 
that Internet surveying invites potentially biased results. The reason for this concern is the 
frequently reported, earlier, findings that the majority of Internet users are young, white, 
educated, males (Gjestland, 1996; Swoboda et al., 1997). However, recent studies report that 
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Internet users are becoming more representative of the population as a whole (Georgia Technical 
Institute’s Graphics, Visualization & Usability Center (GVU), 1999; Kaye & Johnson, 1999). 
Although, the Graphics, Visualization and Usability Center’s tenth World Wide Web User 
Survey (1999) reports most recently a decrease in the number of female respondents. 

Several studies have looked at the differences between the response rates in traditional, 
paper-and-pencil surveys and electronic formats. Most of these studies, particularly the earlier 
studies, have focussed on the use of e-mail as a survey instrument (Heflich & Rice, 1999; 

Schuldt & Totten, 1994). Often, when examining the differences in response rates, the results 
have been mixed. Schuldt and Totten (1994) report their mail survey far out performing their e- 
mail survey, regarding response rates, 56.5% to 19.3% (p. 4). However, in 1997, Smith 
examines the findings of four previous studies and reports one instance in which the e-mail 
response rate exceeded the postal mail rate, one study where the e-mail response rate was much 
lower, and two studies where no significant differences in response rates were reported. 

A second methodology frequently reported concerns on-line surveying where a sample is 
not drawn from a known population. Rather, participants are reached by posting notice of an on- 
line survey via listservs, news groups, and/or other Web pages (Kaye & Johnson, 1999; Smith, 
1997; Smith & Leigh, 1997; Swoboda et al., 1997). These individuals are then given the URL of 
the actual survey instrument and invited to participate in the study. A particular challenge with 
this methodology is to estimate the total number of individuals in your sample. Without this 
knowledge, an exact response rate is difficult to calculate (Smith, 1997, p. 8). 

Swoboda et al. (1997) conducted a hybrid of these two methodologies. They randomly 
selected 200 of 8,017 newsgroups focussing on issues relevant to their desired sample. Next, e- 
mail addresses were acquired from those newsgroups. The sample was then e-mailed the survey 



and invited to participate. Overall, they achieved a 19.3% response rate from this e-mailed 
sample. 

Concerns over how representative data from an e-mail survey may be have also been 
addressed. Smith and Leigh (1997) report that analysis of responses from male participants did 
not show any significant differences across the method of the survey taken. Similarly, female 
responses, whether from the e-mail group or the paper-and-pencil group did not differ. They 
conclude, “obtaining similar patterns of responses, despite these differences in sample 
population, subject selection, survey administration, and testing environments, strongly argues in 
favor of the generalizability and validity of data collected from Internet subjects and hence the 
utility of using the Internet as an alternative or supplemental source of subjects” (Smith & Leigh, 
1997, p. 502). Heflich and Rice (1999) also found their e-mail survey to be “a viable method for 
obtaining reflective data from participants.” 

From the review of literature, several points became clear. First, very few of the studies 
on electronic surveying concerned higher education. The majority of the studies focussed either 
on applications for private business or use in marketing research. Second, no study was found 
where an examination was conducted between paper-and-pencil surveys and an on-line survey 
sent to a specific, targeted sample. Either the format was e-mail, not on-line, or the Web survey 
was open to members of a listserv, newsgroup, etc. where little was known about the potential 
participants and m i ni mal screening was possible. 

This study was developed to compare the benefits and costs associated with on-line and 
paper-and-pencil surveys for a sample that had been purposefully drawn from a known 
population of college undergraduates. The study incorporates elements of quantitative and 
qualitative methodologies that provide different perspectives for examining the critical questions 
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which still remain unanswered: (1) is the profile of Web respondents different, (2) do response 
rates differ, (3) are the results obtained from Web respondents different, and (4) is there less 
burden on respondents with a Web-based survey? 



Methodology 

A closed sample of college undergraduates was randomly selected from the authors’ 
institution. The instrument issued was a locally designed College Selection Survey that asked 
students to evaluate the soundness of ten different reasons for choosing, or not choosing, to 
attend the university, if they had the opportunity to make their choice over again. Half of the 
sample was contacted by mail and asked to complete a paper-and-pencil version of survey. The 
other half of the sample was contacted by e-mail and asked to respond to the same set of 
questions on-line. Both sets of students were also asked whether they would be willing to 
participate in focus groups to discuss how influential the survey reasons would be if they could 
select their college over again. The willing students were contacted and two focus group 
discussions were conducted. The remainder of this section presents the details of the design and 
analysis of the surveys, the focus groups and the sample selection process. 

The College Selection Survey 

The College Selection Survey that was used as the basis for comparing the benefits and 
costs associated with on-line and paper-and-pencil surveys, was developed to illuminate 
students’ reasoning for choosing to attend the college if they had the opportunity to make the 
decision over again. The thinking behind students’ college selection decisions has become a 
critical recruitment and retention issue at the researchers’ institution over the last several years 
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because the percentage of students who affirmed that “they would choose the institution again” 
on state-mandated surveys has been below the state average. 

The College Selection Survey was based on students’ responses to an open-ended prompt 
on a locally developed student satisfaction survey that was administered to a randomly selected 
sample of undergraduates in the spring of 1999. The open-ended prompt invited students to 
explain their reasons for choosing or not choosing the institution again. A content analysis of the 
students’ written responses produced several distinct and valid sets of reasons that appeared to 
motivate students’ college re-selection decisions. These included: (1) students’ sense that they 
are connected to, valued, and treated with dignity and respect on campus; (2) the availability of 
students’ chosen majors and courses; (3) the quality of the faculty and the academic climate; and, 
(4) the comparative cost, appearance, and location of the institution. Ten items were constructed 
based on these responses. In the final form of the survey, after indicating whether they would 
choose the university again, students evaluated the ten items on a five point Likert scale that 
weighed how relevant each reason was to their stated choice. Additionally, students indicated 
whether they would be willing to participate in focus groups; how long they were willing to 
spend on surveys; and whether they preferred to take surveys on paper or on-line. They were 
also invited to elaborate on their reactions to the survey. Students entered their social security 
numbers so that non-respondents and potential lottery winners could be identified, as well as to 
ensure that only those in the sample responded and to prevent multiple responses. 

Sample Selection and Survey Administration 

A random sample of 3,000 undergraduates was selected to receive the College Selection 
Survey. Each survey was accompanied by a brief letter that explained the purpose of the survey, 
insured the confidentiality of the student’s responses, and alerted the students to a $100 cash 
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prize that they would be eligible for if they completed the survey and returned it. 1,500 of the 
students received the letter and the survey which, combined, fit on one side of an 8I/2" by 11" 
sheet of paper. In all, 280 students completed and returned their surveys after the first mailing. 
The non-respondents were mailed a second survey. The second mailing produced additional 207 
surveys, yielding a final response rate of 33.3%. Thirty-six of the 1,500 paper-and pencil 
surveys were returned undeliverable. 

The other half of the sample received notification of the on-line survey via e-mail and a 
post card. Both notifications explained the purpose of the survey, insured confidentiality, offered 
the cash incentive and provided students with the Web address for the survey. When students 
went to the Web-address, they encountered an initial screen that asked for their social security 
numbers. Self-identification was necessary to insure that only the sampled students had access 
to the survey, and to prohibit students from completing the survey more than once. The initial 
contact yielded 256 responses. Non-respondents were sent a second e-mail, which produced 
another 56 responses. A third, and final, e-mailing produced 80 additional surveys, for an 
overall response rate of 26.2%. Three of the 1,500 students had both undeliverable e-mail and 
mail addresses. 

Focus Groups 

A total of 1 84 students from the paper-and-pencil and on-line groups volunteered to 
participate in follow-up focus groups. 135 of these students had responded that they would 
choose the institution again, 16 had responded that they would not, and 33 were unsure. All of 
the students were contacted by phone and invited to participate in a late afternoon discussion 
(pizza and sodas were provided). The fourteen students who consented met in two groups. The 
first group (8 students) was comprised of students who had said they would choose the university 



again, and the second group (6 students) was comprised of students who had said they either 
would not or were unsure. The interviews lasted an hour and a half. The bulk of the time was 
allotted to gaining deeper insight into the students’ reasons for their choices, thus providing 
further validation for the survey items. The fmal fifteen minutes were devoted to questions 
concerning students’ reactions to both versions of the survey, the $100 incentive, and their 
preferences for future surveys. 



Results 

The demographic characteristics of the students who responded to the survey on-line and 
those who responded via the Web were obtained from institutional databases. The composition 
of the two groups, based on class level, major, race, and gender were representative of the 
undergraduate population. 

The overall response rate for the paper-and-pencil sample of 33.3% was significantly 
greater than the 26.2% response rate for the on-line sample (based on a t-test, p < 0.0014 1 ). The 
two samples were similar to each other with respect to sex, race, class, and housing status, but 
they differed with respect to age. The on-line sample had a significantly higher proportion of 
“traditional” students, age 18 to 24 (Figure 1). 

The students in the two groups were similar with regards to whether they would choose 
the institution again. Two-thirds (66.7%) answered “yes,” 8.9% answered “no” and 24.3% said 
they were “not sure.” These results are consistent with findings from previous surveys that asked 
the same question. 



1 The a-level for our 7 comparison tests was adjusted downwards to insure that the overall risk of incorrectly 
identifying a difference between the on-line and paper-and-pencil groups remained at 0.01. 
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Figure 1. Age of respondents by survey method completed. 
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A factor analysis was used to confirm whether the 10 reasoning items could be regrouped 
into the three conceptual scales that were identified during the analysis of the earlier survey 
results. The procedure reproduced the three factors. The first of these, Academics, had high 
factor loadings (>.50) for the availability of the students’ major, course convenience, quality of 
the faculty and academic reputation. The second factor, Feeling Valued, had high loadings 
(> .76) for feeling connected, feeling valued and treated with dignity and respect. The final 
factor, Campus Attributes had high loadings (> .78) for campus appearance and location. Scores 
for all three factors were obtained by averaging the relevant items and a manova was used to 
determine if there were any differences in the factor scores between the two groups. The results 
indicated that there were no differences. 
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A significant difference was found in the percentages of students who volunteered written 
comments (%i 2= 14.8 p=0.0001). Over one-third of the Web respondents (37.5%) wrote 
comments compared to 25.5% of the paper-and-pencil respondents. The content of the students’ 
comments was evaluated on a 5-point rating scale. “Very Favorable” comments received a score 
of 5 and “Very Unfavorable” comments received a score of 1 . A t-test was used to determine if 
there were differences in the favorability of the comments between groups. The test showed that 
students in the on-line group were significantly more favorable (mean=3.2, sd=1.6) than students 
in the paper-and-pencil group (mean=1.6, sd=1.6). These results are presented in Figure 2. 



Figure 2. Open-ended comments by survey method completed. 
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Significant differences were also found between the groups regarding whether students preferred 
taking surveys on paper or on-line. An overwhelmingly greater proportion of students (y^ 2 = 
321.3 p=0.001) from the on-line group stated a preference for on-line surveys (83.7% compared 
to 23.6%, as depicted in Figure 3). 



Figure 3. Preference for format a future surveys by survey method completed. 
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The advantages and disadvantages of on-line surveying were further explored in the focus 
group discussions. From the perspective of those students who are comfortable with and have 
access to the technology, the biggest advantage of on-line surveying is its convenience. Students 
can sit down and complete the survey, without having to handle paper, place it in an envelope, 
find a stamp and transport it to a mailbox. On the other hand, students who do not have direct 
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access to a computer complained about the difficulty of finding open computers in the campus 
laboratories. Computers are often unavailable when students go to the labs, and once they gain 
access, they are worried about task deadlines and do not take time to respond to e-mail or 
surveys. Students also complained about the awkwardness of their on-campus e-mail accounts 
and told us that they preferred to use their own Web-based accounts (i.e., Hotmail.com). There 
were also students who described themselves as “technologically illiterate” and talked about their 
fear and intentional avoidance of computers. 

Finally, we explored ways that on-line surveys could be improved. In response to the 
survey question that asked how long students were willing to spend on a survey the mean 
reported response time was 13 minutes (sd = 13.4 minutes). There were no significant 
differences in the reported times across survey method or students’ expressed preference for 
future survey formats. 

In the focus groups, we asked students if they had been motivated to complete the survey 
by the $100 lottery prize. Some students had not been particularly enticed by the chance for the 
monetary reward and sincerely responded out of a sense of duty to the university. Others had 
overlooked the incentive entirely. When asked about the types of incentives that would be more 
enticing, students suggested movie tickets and gift certificates that could be redeemed at local 
shopping malls and restaurants. 
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Discussion 



Taken as a whole, it appears that the demographic and attitudinal profiles of the on-line 
respondents were not significantly different from the paper-and-pencil respondents. Both 
methodologies were representative of the undergraduate population, while no significant 
differences appeared across methodologies with respect to sex, race, class, housing or 
evaluations of the university. The only difference occurred when examining the methodologies 
by age group. 

Perhaps the higher percentage of traditional age students responding on-line is a sign of 
growing enthusiasm and comfort with on-line surveys. The GVU’s tenth World Wide Web User 
Survey reports an interesting profile for the number of years respondents have been on the Web, 
with the largest category having been on 4-6 years (37.1%). Previously, the largest category had 
been on-line for 1-3 years. This indicates a shift toward more experienced users (on average) 
and may explain the higher response rate among students in the 18 to 24 age group (GVU's tenth 
WWW user survey, 1999). As familiarity with technology continues to expand, perhaps an 
increased comfort and willingness to take on-line surveys will arise. 

Web respondents appeared more comfortable with supplying comments than did the 
other participants. Additionally, the on-line comments on average were more favorable. The 
greater percentage of on-line respondents supplying comments appears to support Davis’ (1997) 
finding that this method may add an additional sense of confidentiality over conventional paper- 
and-pencil methods. However, it is unclear as to why the on-line format elicited generally more 
positive comments. 

Surveys, regardless of format, need to be brief. On average, students who completed the 
survey from our sample stated they were willing to spend 10-15 minutes on a survey of this type. 
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Most students prefer on-line surveys for several reasons or don’t have a preference on format. It 
is encouraging to see that most feel comfortable with the Web format. 

Response rates could be increased with incentives that are immediately tangible to 
students, such as movie tickets or gift certificates. According to those in the focus groups, 
monetary incentive was not necessarily a motivating factor to reply. 

There is a need to search for alternative ways of contacting students to complete on-line 
surveys other than university e-mail accounts since many students have separate e-mail through 
their Internet service providers and do not forward their university accounts. When contacting 
students via mail, it is often difficult to determine which address to use. Most students have a 
campus mailbox, but not all check it if they live locally. 

Some of the limitations of this study include the make up of the student body, type of 
institution, and focus of the survey instrument. More research is needed before these findings 
can be generalizable to other populations. The higher response rate for the paper-and-pencil 
method may be due to greater comfort with this format, or possibly a lack of convenient access 
to technology. Further studies are needed to examine what factors enter into the preference of 
one format over another and how that impacts survey response rates. Additional studies could 
continue to examine whether or not future freshmen classes will show a greater comfort with 
technology. Also, what impact taking a survey on-line has on the frequency and tone of 
comments would provide additional insight for surveying professionals. 

Despite these limitations, it may be concluded that surveying on-line has the potential to 
be, in fact, both an efficient and reliable means by which to poll students. 
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